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The present invention relates to interpolating missing color values of a pixel in 
an array of pixels. 

In digital cameras a lens is used to obtain light information from a scene. 
5 These light photons are converted into electrons by an image sensor consisting of a number 
of light sensitive elements. When using digital RGB still cameras, three image sensors are 
used to obtain color sensing for each color red, green and blue. In order to reduce both cost 
and size of the camera, it is possible to use one image sensor having an RGB Bayer filter 
array in which each pixel in the sensor array senses red, green or blue in a predefined pattern. 

10 This pattern is build up of alternating green, red columns and green, blue columns. When 
using one image sensor to sense all three colors, it is necessary to reconstruct the missing 
pixels before the total image is represented. 

In WO 99/04555 a green reconstruction method for RGB Bayer image sensors 
has been described. The reconstruction concerns only the green color reconstruction, the red 

15 and blue colors remain reconstructed in a conventional way. The reconstruction of the 
missing green pixel is carried out by means of a median filter that sorts three specific 
variables: two of them are derived from the green color, the third one from the red or blue 
color. A disadvantage of this method is that for high saturated colored edges artefacts which 
look like the border of a postage stamp are introduced. 

20 First a short description of the green reconstruction method of WO 99/04555 

will be given. This method will be referred to as smartgreenl reconstruction. The algorithm 
for smartgreenl is based on the fact that resolution losses are best observed in high frequency 
near white scene parts and less near colored parts. 

With this in mind, the contribution of the red and blue pixel is used to help 

25 determine the reconstruction value of the missing green pixels. The objective of smartgreenl 
reconstruction is to maximize the resolution of the green color. For this purpose a median 
filter algorithm is applied. In Fig. 1 the chosen definition of the green pixels surrounding the 
missing green pixel, i.e. the location which is occupied by a red or blue (R/B) pixel, is shown. 
This is used to declare the following variables: 
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G12 = (Gl+G2)/2 
G34 = (G3+G4)/2 

Further the variable RBc is defined as the near white adapted red or blue color 
value of the center pixel, defined later, and SmartGcntrlR and SmartGcntrlB are near white 
5 matrix and white balance correction parameters. The calculation of these parameters is 
described below. 

In smartgreenl reconstruction for especially digital signal, processing is 
performed by sorting the data of G12, G34 and RBc in order of magnitude. The center value 
of the sorted sequence, also called the median value, will be applied for the reconstruction of 

10 the missing green pixel. 

smartG = median( G12, G34, RBc) 

For a red and blue center pixel, respectively, the RBc value becomes: 
RBc = SmartGcntrlR * R 
RBc = SmartGcntrlB * B 

15 The median filter for green replaces the conventional green reconstruction. 

The conventional red and blue reconstruction method is, however, maintained. 

In Fig. 2 an example of a vertical green-magenta scene color transient is 
shown. In total four different modes are illustrated: A trailing green edge and a leading green 
20 edge and the phase relations to the missing green pixel in the center. 

The green color is illustrated at the top of the Fig.; above the pixel definitions 
and below the definitions the magenta transients are shown. Finally, the corresponding 
projection of green on the imaginary sensor pixels is shown when interpolating using the 
smartgreenl method. 

25 According to the smartgreenl median algorithm, being smartG = median(G12, 

G34, RBc), the result of the reconstructed center green becomes 0.5 for all four modes. 

For the shown vertical colored edges smartgreenl causes a green intensity 

modulation in the vertical direction resembling the border of a postage stamp. It should be i 

noted that the same applies to colored edges in the horizontal direction. Fig. 3 shows the | 

i 

30 RGB reconstruction result with smartgreenl of a green-magenta scene transient. Green to the 
left and magenta to the right. 

It is an object of the present invention to provide a method for green color 
reconstruction preventing the above artefacts. To this end, the invention provides a green 
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reconstruction for image sensors as defined in the independent claims. Advantageous 
embodiments are defined in the dependent claims. 

According to a preferred embodiment of the present invention, the green 
modulation problem can be prevented using a median filter that sorts the four individual 
5 neighboring green pixels and the center RBc value. The following applies to this 5-pixel 
median filter: smartG = median(Gl, G2, G3, G4, RBc) 

In a preferred embodiment of the present invention, it is determined whether a 
selected pixel forms part of an edge in the image and whether said edge has an angle to the 
vertical line substantially different from 90 degrees. If so, the prior art smartgreenl algorithm 
10 is used, while the inventive 5-input median is used for substantially vertical edges. The edge 
detection preferably involves the steps of: 

a) forming a first pair of diagonal pixels by selecting a vertical and a horizontal 
neighboring pixel and determining the difference between the color values of these pixels, 

b) forming a second pair of diagonal pixels by selecting a vertical and a 
15 horizontal neighboring pixel and determining the difference between the color values of these 

pixels, and 

c) determining the difference between color values of the pixels of the first and 
second pair of pixels. 

Step c) may further comprise a step of determining if the determined 
20 differences are below the predefined level. This predefined level may be between 3 and 10 

percent of maximum color value, such as between 4 and 8 percent of maximum color, such as 
between 5 and 7 percent of maximum color value. 

These and other aspects of the invention will be apparent from and elucidated 
with reference to the embodiments described hereinafter. 



25 



In the drawings, 

Fig. 1 shows the definition of the green pixels surrounding the missing green 



pixel; 



Fig. 2 shows a vertical green-magenta scene color transient; 
30 Fig. 3 shows the RGB reconstruction result with smartgreenl of a green- 

magenta scene transient; 

Fig. 4 shows the colored edge reproduction by the 5-pixel median filter; 
Fig. 5 shows the reconstructed RGB transient with the 5-pixel median filter; 
Fig. 6 shows the diagonal direction detection; 
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Fig. 7 shows an RGB Bayer luminance pixel; 

Fig. 8 shows the RGB and contour reconstruction, matrix and white balance of 
an RGB Bayer color camera; and 

Fig. 9 shows a block diagram for parallel RGB reconstruction and contour 

5 filtering. 

With a green-magenta colored scene transient, the green reconstruction result 
of this filter becomes as shown in Fig. 4. The green color is illustrated at the top of the Fig. 
and between the imaginary pixel illustrations the magenta transients are shown above the 
10 corresponding projection of green on the imaginary sensor pixels. The undesired modulation 
at the colored edges has vanished. In Fig. 5 the RGB reconstruction result of the 5-pixel 
median filter is shown. Green is shown on the left side of the Fig. and magenta is shown on 
the right side. 

Unfortunately, the 5-pixel median filter causes some extra distortion in a zone 
15 plate scene, especially in the diagonal direction just outside the green Nyquist domain. It also 
suffers from distorted border artefact if the colored edges are rotated 45 degrees. A direction 
dependent condition can help to reduce or eliminate the diagonal amount of distortion. 

A diagonal direction detection can be performed using the following 

algorithm: 

20 If a first set of two diagonal pixels has approximately the same first green 

value and a second set of diagonal pixel has approximately the same second green value 
being different from said first green value, there is a large chance that it is a diagonal edge, 
see Fig. 6. Written in a software form this becomes: 
if ((abs(Gl-G3)<SGllevel) and (abs(G4-G2)<SGllevel)) xor 

25 ((abs(G3-G2)<SGllevel) and (abs(Gl-G4)<SGllevel)) then select smartgreenl 
else select 5-pixel median filter. 

A certain percentage (5-7 percent) of the full-scale amplitude applies to the 
SGlevel. In practice it is about 6.25% (being a value of 16 on a 256 full-scale range). 
30 Referring to the previous declaration of variables as applied for smartgreenl, 

the following applies to the new green reconstruction method according to this invention: 
if ((abs(Gl-G3)<SGllevel) and (abs(G4-G2)<SGllevel)) or 
((abs(G3-G2)<SGllevel) and (abs(Gl-G4)<SGllevel)) then 
smartG = median( G12, G34, RBc ) 



else smartG = median( Gl, G2, G3, G4, RBc) 
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To the RBc value applies that: 
RBc = SmartGcntrlR * R 
5 RBc = SmartGcntrlB * B 



This new method has been tested on colored edges and has been compared 
with the conventional reconstruction. Clearly, the method of the present invention offers the 
best results. 

10 In the following the above mentioned parameters SmartGcntrlR and 

SmartGcntrlB will be explained. 

For a near white luminance signal in the reconstruction block, derived from 

the RGB pixels of the image sensor, the matrix and white balance parameters have to be 

taken into account. Since the matrix and white balance are located after the reconstruction, 
15 some adaptation of the incoming red and blue colors is necessary. For that purpose the 

parameters SmartGcntrlR and SmartGcntrlB are used to control the red and blue amplitudes 

in order that they match with green and result in a near white luminance signal Yn. Referring 

to Fig. 7, the following applies to this Yn-signal in case of red and blue pixels: 

Yn[i j] = SmartGcntrlR * red 

20 Yn[i+l,j+l] = SmartGcntrlR * blue 

In case of green pixels Yn is equal to green: 

Yn[i+l,j] = green 

Yn[ij+1] = green 



25 

In Fig. 8 a simplified block diagram is shown with the RGB and aliasing free 
contour reconstruction, followed by the matrix and the white balance. This block diagram is 
used to define the parameters in the next formulas for the calculation of SmartGcntrlr/B. 
Light LS from a scene is passed to an RGB Bayer sensor S thru a lens L. An output signal 
30 from the sensor S is applied to a CDS (correlated double sampling, age (automatic gain 
control) and ADC (analog to digital conversion) processing block 1. An output of the 
processing block 1 is applied to an RGB reconstruction and parallel contour processing block 
3. The processing block 3 outputs reconstructed RGB signals Ri, Gi and Bi, as well as an 
aliasing-free contour signal AFC. The reconstructed RGB signals Ri, Gi and Bi are applied to 
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a matrix circuit MX that produces signals Ro, Go and Bo, which are applied to a white 
balance circuit WB to furnish output signals Ro', Go' and Bo'. 

A correction of each RGB Bayer color sensor's primary colors into the EBU 
5 primaries, which are accustomed in worldwide television sets and PC monitors, is necessary. 
The correction is realized with a matrix that requires nine multipliers. 
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Ro, Go, Bo are the output RGB signals of the matrix and Ri, Gi, Bi are the input signals. 

10 With the white balance after the matrix, the RGB signals become: 

Ro' = awbR ■ Ro 

Go' = Go 

Bo' = awbB • Bo 

where awbR and awbB are the white balance parameters. (According to the World Gray 
Assumption method (WGA), awbR=totalGreen/totalRed and awbB=totalGreen/totalBlue, 
where totalRed, totalGreen and totalBlue represent the total of the RGB color amplitudes 
15 measured over the whole scene.) Both actions, the matrix together with the white balance, 
offer the desired white reproduction. The Ro', Go', Bo' signals now guarantee an EBU color 
reproduction. 

For a proper near white luminance signal Yn the opposite has to be done. 
20 Therefore, imagine a scene with colors according to the EBU color space and a color 

temperature equal to D65 white. With the inverse matrix of the one shown below, the color 
space of the sensor is achieved: 
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where Rii, Gii, Bii represent the colors of an EBU scene and Ri,Gi, Bi the colors of the ! 
25 sensor. 

For the luminance signal Yn only the white reproduction of the inverse matrix 
is of interest, being represented by the sum of the matrix coefficients of each color. 
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ZRii = Ml + M2 + M3 
ZGn = £21 + £22 + £23 
LBii = £31 + £32 + £33 



In addition, the color temperature of the scene need not be D65 white. 

Inclusive an arbitrary color temperature the sum of the matrix coefficients becomes: 

Z/?nv£ = Rpresetgain • M 1 + Gpresetgain • £12 + Bpresetgain • M3 
5 EGiw£ = Rpresetgain • £21 + Gpresetgain • £22 + Bpresetgain • £23 
UBiwb = Rpresetgain • £3 1 + Gpresetgain * £32 + Bpresetgain • £33 



where Xpresetgain (X=R, G or B) represents the gain factors for transferring D65 white into 
that arbitrary color temperature. (For D65 white all Xpresetgain parameters are one.) 
To the SmartGcntrlR/B parameters used in Yn[i,j] and Yn[i+l,j+l] (see formula below) 
10 applies that: 

XGiwb 



SmartGcntrlR = 
SmartGcntrlB = 



lUiwb 
XGiwb 



ZBiwb 

The parameter ZGiwb is used as nominator because the green amplitude is 

regarded as a reference. This applies to the white balance as well. 

Now the above formulas can be written in such a way that the measured white 

15 balance parameters awbR/B may be applied. Knowing that 

Gpresetgain f _ Gtotal 

— = awbR = 

Rpresetgain Rtotal 

Gpresetgain _ _ Gtotal 

Bpresetgain Btotal 

therefore 

„. . ^ . ( Ml ,- t M3 ^ 

Zifiw£ = Gpresetgain • + Z>21 + 

^aw£R aw£Z? J 

(£21 £23 ^ 
+ £22 + 
awbR awbB > 

VI> . y _ . ( £31 LOO £33 \ 

ItBiwb = Gpresetgain • + £23 + 

^aw£R a>v££ J 
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Since the ZXiwb- values are divided in the above, the parameter Gpresetgain is 
not important because Gpresetgain/Gpresetgain=l. Therefore, the next formula is sufficient 
for calculating the desired XXiwb-values: 



^ awbR awbB 

f 621 623 
+ 622 + 



ZGiwb = 



1 



^awbR awbB J 

„. _ ( Ml tM £33 \ 
YBiwb = I + 2>23 + 

5 

Now a luminance signal Yn has become available with equal RGB signal 
amplitudes for white scene colors due to the fact that the sensor matrix and the color 
temperature of the scene have been taken into account. 

10 Fig. 9 shows the block diagram of the RGB reconstruction. Yn is the 

multiplexed RGB-signal of the sensor where R has been multiplied with SmartcntrlR, and B 
with SmartcntrlB, in a preprocessing block 5. This Yn-signal is used for the embodied 
smartgreen reconstruction. Via a zero switchbox ZSB, Yn is splitted into three colors, red = 
R*SmartcntrlR, green = G, and Blue = B*SmartcntrlB, making the conventional Laplacian 

15 RGB reconstruction possible with smart green in a 3x3 RGB reconstruction with/without 
smartgreen processing block 7. The so-called RBc signal in the median filter already fits 
R*SmartGcntrlR and B*SmartGcntrlB. 

By dividing the reconstructed red and blue signals in dividers Dr and Db by 
SmartGcntrlR and SmartGcntrlB, respectively, the original red (Ro) and blue (Bo) sensor 

20 amplitudes are restored. This means that the usually applied matrix, white balance and 
gamma functions can be maintained. In digital circuit design multipliers are preferred to 
dividers. Therefore, in order to avoid the divider circuits, the best way is to let the computer 
of the camera calculate 1/SmartcntrlR and 1/SmartcntrlB. Next, via two separate wires, those 
values can be offered to two multipliers. The Ro-amplitude now becomes equal to the R- 

25 amplitude of the input signal (SmartcntrlR*R*(l/SmartcntrlR=R). The very same applies to 
the Bo-amplitude. 

It should be noted that the parameters SmartcntrlR/B have been determined in 
a measurement cycle before the photograph is taken or in a continuous way in case of video 
mode. 
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The image sensor used for generating the array of pixels may form part of a 
digital camera. The pixels of the array may be arranged in perpendicular rows and columns; 
however, alternative arrangements may also be possible. 

In order to obtain full color images with high color resolution, the sensor may 
be covered with a mosaic-like color pattern, where each pixel of the sensor is positioned 
behind a given color filter - e.g. a green filter. The RGB Bayer mosaic color filter would be a 
preferred choice in the present invention. However the method could also be used with a 
YeGCy Bayer sensor. Then the SmartGcntrlR/B parameters are calculated as where Ye-G = 
Red and Cy-G = Blue: 

ZGiwb 



SmartGcntrlR = 
SmartGcntrlB = 



JJRiwb + lLGiwb 

ZGiwb 
ZBiwb + XGiwb 



Although the present invention has been described in connection with the 
preferred embodiment, it is not intended to be limited to the specific form set forth herein. On 
the contrary, it is intended to cover such alternatives, modifications, and equivalents, as can 

15 be reasonably included within the scope of the invention as defined by the appended claims. 
In the claims, any reference signs placed between parentheses shall not be construed as 
limiting the claim. The word "comprising" does not exclude the presence of elements or steps 
other than those listed in a claim. The word "a" or "an" preceding an element does not 
exclude the presence of a plurality of such elements. The invention can be implemented by 

20 means of hardware comprising several distinct elements, and by means of a suitably 

programmed computer. In the device claim enumerating several means, several of these 
means can be embodied by one and the same item of hardware. The mere fact that certain 
measures are recited in mutually different dependent claims does not indicate that a 
combination of these measures cannot be used to advantage. 
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1. A method of interpolating a missing color value of a pixel, said pixel forming 

part of an array of pixels generated by an image sensor having a color filter array, said array 
of pixels comprising a first and a second group of pixels, wherein the first group of pixels 
represents parts of an image optically filtered with a filter having a first color (G), and 
5 wherein the second group of pixels represents parts of the image optically filtered with one or 
more filters having one or more second colors (R, B), each pixel of the second group having 
a vertical and a horizontal neighboring pixel of the first group, said method comprising: 

providing individual first color informations of all neighboring vertical and 
horizontal pixels, 

10 selecting a pixel of the second group and providing second color information 

about the one or more second colors at the position of the selected pixel, and 

interpolating the missing color value at the position of the selected pixel using 
the individual first color informations and the second color information, said interpolating 
step comprising the step of calculating a median value of the individual first color 

15 informations and the second color information. 



2. A method according to claim 1, wherein said interpolating step comprises: 

determining whether the selected pixel forms part of an edge in the image and 
whether said edge has an angle to a vertical line being substantially different from 90 
20 degrees, said vertical line being defined by the neighboring vertical pixels to the selected 
\ pixel, and 

in case the edge has an angle to the vertical line being different from 90 degrees: 

determining a horizontal average value of the horizontal neighboring pixels, 
determining a vertical average value of the vertically neighboring pixel, 
25 calculating a median value using the determined horizontal average value, the 

determined vertical average value and the second color information at the position of the 
selected pixel, and 

in case the edge has an angle to the vertical line substantially equal to 90 degrees: 



e 
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calculating the median value of the individual first color informations and the 
second color information. 

3. A method according to claim 2, where the step of determining whether the 

5 selected pixel forms part of an edge in the image and whether said edge has an angle to the 
vertical line substantially different from 90 degrees comprises: 
. a) forming a first pair of diagonal pixels by selecting a vertical and a horizontal 

neighboring pixel and determining the difference between the color values of these pixels, 

b) forming a second pair of diagonal pixels by selecting a vertical and a 
10 horizontal neighboring pixel and determining the difference between the color values of these 

pixels, and 

c) determining the difference between color values of the pixels of the first and 
second pair of pixels. 

15 4. A method according to claim 3, wherein each of steps a) and b) further 

comprises a step of determining if the determined differences are below a predefined level. 

5. A method according to claim 4, wherein step c) further comprises a step of 
determining if the determined differences are below the predefined level. 

20 

6. A method according to claim 4 or 5, wherein the predefined level is between 5 
and 7 percent of maximum color value. 

7. A computer program comprising computer program code means adapted to 
25 perform the method according to claim 1 when said computer program is run on a computer. 

8. A computer readable medium carrying the computer program according to 
claim 7. 

30 9. A device for interpolating a missing color value of a pixel, said pixel forming 

part of an array of pixels generated by an image sensor having a color filter array, said array 
of pixels comprising a first and a second group of pixels, wherein the first group of pixels 
represents parts of an image optically filtered with a filter having a first color (G), and 
wherein the second group of pixels represents parts of the image optically filtered with one or 
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more filters having one or more second colors (R, B), each pixel of the second group having 
a vertical and a horizontal neighboring pixel of the first group, said method comprising: 

means for providing individual first color informations about all neighboring 
vertical and horizontal pixels, 

means for selecting a pixel of the second group and providing information 
about the one or more second colors at the position of the selected pixel, and 

means for interpolating the missing color value at the position of the selected 
pixel using the individual first color informations and the second color information, said 
interpolating means comprising means for calculating a median value of the individual first 
color informations and the second color information. 

10. A color camera, comprising: 

a sensor (S) having a color filter array; and 
a device (3, 7) as claimed in claim 9. 
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The present invention relates to interpolating a missing color value of a given 
pixel in an array of pixels. The missing color value is determined from color values of 
neighboring pixels using a median-based technique in which the median is taken of the four 
pixels values of the horizontally and vertically neighboring pixels (G1-G4) having a same 
color as the missing color value, and color information from at least one other color (R/B) at 
the given pixel. 



(Fig. 1) 
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